Bayesian Neural Networks for Industrial Appli

Aki Vehtari and Jouko Lampinen Laboratory of Computational Engineering, Helsinki University of Te hnology P.O.Box 9400, FIN-02015 HUT, FINLAND SMCia/99 1999 IEEE Midnight-Sun Workshop on Soft Computing Methods in Industrial Appli ations Kuusamo, Finland, June 16 18, 1999 Abstra t We demonstrate the advantages of using Bayesian neural networks in regression, inverse and lassi ation problems, whi h are ommon in industrial appli ations. The Bayesian approa h provides onsistent way to do inferen e by ombining the eviden e from data to prior knowledge from the problem. A pra ti al problem with neural networks is to sele t the orre t omplexity for the model, i.e., the right number of hidden units or orre t regularization parameters. The Bayesian approa h o ers e ient tools for avoiding over tting even with very omplex models, and fa ilitates estimation of the on den e intervals of the results. In this ontribution we review the Bayesian methods for neural networks and present omparison results from ase studies in predi tion of the quality properties of on rete (regression), ele tri al impedan e tomography (inverse problem) and forest s ene analysis ( lassi ation). The Bayesian networks provided onsistently better results than other methods. I. Introdu tion In lassi ation and non-linear fun tion approximation neural networks have be ome very popular in reent years. With neural networks the main di ulty is in ontrolling the omplexity of the model. Another problem of standard neural network models is the la k of tools for analyzing the results ( on den e intervals, like 10 % and 90 % quantiles, et .). The Bayesian approa h provides onsistent way to do inferen e by ombining the eviden e from data to prior knowledge from the problem. Bayesian methods use probability to quantify un ertainty in inferen es and the result of Bayesian learning is a probability distribution expressing our beliefs regarding how likely the di erent predi tions are. Predi tions are made by integrating over the posterior distribution. In ase of insu ient data the prior dominates the solution, and the e e t of the prior diminishes with in reased eviden e from the data. For neural networks, Ma Kay introdu ed Bayesian approa h [1℄ based on Gaussian approximation. Neal has introdu ed hybrid Monte Carlo method [2℄ that failitates Bayesian learning for neural networks with no ompromising approximations. The main advantages of Bayesian neural networks are: • Automati omplexity ontrol: Bayesian inferen e te hniques allow the values of regularization oe ients to be sele ted using only the training data, without the need to use separate training and validation data. • Possibility to use prior information and hierar hi al models for the hyperparameters. • Predi tive distributions for outputs. In this ontribution we demonstrate the advantages of Bayesian neural networks in three ase problems. First we brie y reviewMulti Layer Per eptron network in se tion II and in se tion III we give a review of the Bayesian methods for neural networks. In se tion IV we present results using Bayesian neural networks in a regression problem for predi ting the quality properties of on rete. In se tion V we present results using Bayesian neural networks to solve inverse problem in image re onstru tion and void fra tion estimation in ele tri al impedan e tomography. Results omparing Bayesian neural networks and other lassi ation methods for lassi ation of obje ts in forest s enes are presented in se tion VI. II. Multi Layer Per eptron In this se tion we brie y review Multi Layer Pereptron (MLP) neural network. See [3℄ for thorough introdu tion to MLPs. We on entrate here to one hidden layer MLP networks with hyperboli tangent (tanh) a tivation fun tion, but Bayesian methods des ribed an be used for other types of neural networks too. Basi MLP network model with k outputs is fk(x,w) = wk0 + m ∑